Brown Dwarf: A Distributed Data Warehouse for the Cloud

In this paper we present the Brown Dwarf, a distributed system designed to efficiently store, query and update multidimensional data over commodity network nodes, without the use of any proprietary tool. Brown Dwarf manages to distribute a highly effective centralized structure among peers on-the-fly, reducing cube creation and query times by enforcing parallelization. Both point and aggregate queries as well as updates are naturally performed on-line through cooperating nodes that hold parts of a fully or partially materialized data cube. The system also employs an adaptive replication scheme that expands or shrinks the units of the distributed data structure for minimal storage consumption against failures and load skew. Brown Dwarf collects many of the features of an application to be deployed in the Cloud: It adapts its resources according to demand, allows for on-line, fast and efficient storage/processing of large amounts of data and is cost-effective both over the required hardware and software components. Our system has been evaluated on both actual and simulation-based testbeds. To outline the findings of our extensive experimentations, Brown Dwarf manages to accelerate cube creation up to 5 times and querying up to several tens of times by exploiting the capabilities of the available network nodes working in parallel. Incurring only a small storage overhead compared to the centralized algorithm, it distributes the structure pretty evenly across the overlay nodes. It manages to quickly adapt even after sudden bursts in load and remains unaffected with a considerable fraction of frequent node failures. These advantages are even more apparent for dense and skewed datacubes and workloads.